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Forum on Speech-to-Text and Scribing: 
Getting a Handle on What This Means 


Background 

More than 45 individuals representing staff from state departments of education, school 
districts, testing and testing-related companies, and other educational organizations 
participated in a forum on June 26, 2018 in San Diego, California to discuss speech-to- 
text and scribing, the differences in terminology, challenges associated with speech-to- 
text, and needed research. The forum was a pre-session to the Council of Chief State 
School Officers (CCSSO) National Conference on Student Assessment (NCSA), and was 
a collaboration of the Assessing Special Education Students (ASES), the State 
Collaborative on Assessment and Student Standards (SCASS), and the National Center 
on Educational Outcomes (NCEO). 


Specific goals of the pre-session forum were to: (a) begin to identify and discuss the 
issues surrounding speech-to-text and scribing; (b) identify policy needs for speech-to- 
text and scribing; (c) learn about assessment vendor speech-to-text and scribing tools; and 
(d) discuss implementation challenges and ways to address them. Each of these topics 
was considered in light of the needs of general education students,’ students who are 
English learners (ELs), and students with Individualized Education Programs (IEPs) or 
504 plans; these considerations were discussed in breakout sessions. 


Purpose 

The purpose of the forum was to develop greater clarity on the implementation of speech- 
to-text and scribing during instruction and assessments. Its purpose also was to discuss 
the current and emerging issues surrounding the uses of speech-to-text for all students. 
This topic grew out of previous discussions about implementing accessibility frameworks 
for all students and the need for common language to discuss accessibility features and 
accommodations (see Warren, Lazarus, Strunk, & Thurlow, 2017; Warren, Thurlow, 
Christensen, Shyyan, Lazarus, & Chartrand, 2016°). Among the questions addressed in 
the forum were those related to the evidence of the validity of results when speech-to-text 
is used by different students. A major question posed was how states can work with 
vendors to develop speech-to-text systems that more smoothly interface with the test 


1 Students who are in general education classrooms and are not identified as a student with a 
disability or an English learner (EL) are referred to in this report as “general education students.” It 
is recognized that students with IEPs and 504 plans, as well as ELs, are also general education 
students, but for purposes of brevity here, we use the term “general education students” to refer to 
those students who do not have disabilities and are not ELs. 

2 Warren, S. H., Lazarus, S. S., Strunk, K., & Thurlow, M. L. (2017). Forum on text readers for everyone 
on all tests: Getting a handle on what this means. Minneapolis, MN: University of Minnesota, National 
Center on Educational Outcomes. Available at 
https://nceo.umn.edu/docs/OnlinePubs/2017ForumReport.pdf 

3 Warren, S., Thurlow, M., Christensen, L., Shyyan, V., Lazarus, S., & Chartrand, A. (2016). Forum on 
common language for states and assessment vendors. Minneapolis, MN: University of Minnesota, 
National Center on Educational Outcomes. Available at 
https://nceo.umn.edu/docs/OnlinePubs/2016ForumReport.pdf. 


delivery system. Further, what guidance is needed related to student use of their own 
devices for speech-to-text during testing? Related questions included how to address 
potential security issues and when it is appropriate for human scribes to scribe the 
speaking portion of an English language proficiency assessment to produce a transcript. 
Finally, what kinds of resources and training are needed by educators, parents, and 
students to support better decision making and implementation of speech-to-text and 
scribing. 


The pre-session forum started with the presentation of information on current state 
policies on speech-to-text (including the terminology used, and the ways in which 
speech-to-text is treated in policy). Then three perspectives on speech-to-text readers 
were provided: (a) state perspective, (b) assessment consortium perspective, and (c) 
vendor perspective. Following these presentations, attendees broke into groups to discuss 
text readers in relation to three groups of students: 


1. General education students 
2. ELs 
3. Students with IEPs or 504 plans 


Each of these groups addressed several questions, including: 


1. What are the implementation and policy challenges that need to be addressed? 

2. How can states work with vendors to get speech-to-text systems that more 
smoothly interface with the test delivery system? 

3. What research is needed for speech-to-text? 


Although participants had limited time for discussion, the discussions were rich and 
engaging. The agenda for the three-hour forum was as follows: 


Welcome and introduction (Sandra Warren, CCSSO ASES SCASS) 
Overview of issues (Martha Thurlow, NCEO) 
Results of state policy analysis (Sheryl Lazarus and Kathy Strunk, NCEO) 
Issues from three perspectives: 
— States (Andrew Hinkle, Ohio Department of Education; Toni Wheeler, 
Washington Office of Superintendent of Public Instruction) 
— Test Vendor (Jan McSorley, Pearson) 
— Assessment Consortium (Trish Klein, Smarter Balanced) 
e Breakout discussion sessions 
— General education students (facilitated by Sheryl Lazarus, NCEO) 
— ELs (facilitated by Martha Thurlow, NCEO) 
— Students with IEPs or 504 plans (2 sections; one facilitated by Sandra Warren, 
CCSSO; one facilitated by Maureen Hawes, NCEO) 
e Debrief and next steps 


Structure of this Report 

This report summarizes both the introductory information provided to forum participants, 
and the facilitated forum discussions that followed. Summaries of the presentations were 
developed from notes taken during the presentations, and from the presenters’ slides. 
Summaries of the facilitated discussions were developed from notes taken by notetakers. 


Participants were encouraged to comment and discuss freely, with assurances that no 
individual’s name, nor any state, company, or organization would be attached to 
comments that were made. Complete anonymity of statements was ensured. This led to 
frank and open conversations. 


Forum Introduction 

Sandra Warren, CCSSO ASES SCASS Advisor, provided an overview of the forum and 
recognized the hosts—-ASES SCASS and NCEO. She raised questions about speech-to- 
text and scribing, then confirmed that the goal of the forum was to begin to develop 
clarity on the implementation of speech-to-text for all students, and to discuss the issues 
surrounding the use of speech-to-text for all students. 


Forum Overview 

Martha Thurlow, NCEO, highlighted the goals of the session and presented the questions 
that would be addressed during the session. She assured participants of the desire for 
open and candid conversations, and noted that a report would be produced as a result of 
the discussions. 


State Policy Analysis 

Sheryl Lazarus and Kathy Strunk of NCEO presented preliminary results of a state policy 
analysis being conducted by NCEO. They noted that the purpose of the analysis was to 
provide a snapshot of how accessibility features and accommodations that use technology 
to deliver speech-to-text in an assessment are included in state policies, specifically 
focusing on English language arts (ELA) and math assessments for the 2017-18 school 
year. They highlighted the variety of terms that are used in states’ policies (e.g., speech- 
to-text, voice recognition), and summarized the ways in which speech-to-text and 
scribing are presented in state policies (e.g., as accessibility features or accommodations) 
depending on the content (e.g., directions, items, or passages for ELA and mathematics). 
Also addressed were the issues related to the use of a speech-to-text as assistive 
technology device interacting with a test platform. 


Four Perspectives on Text Readers 

State Perspective: Andrew Hinkle, Ohio Department of Education 

Andrew Hinkle noted the importance of the topic by summarizing the exponential 
increase in requests for speech-to-text in Ohio, from zero requests in 2015 to 67 requests 
for speech-to-text in 2018. For a student to be able to use speech-to-text, a school 


representative has to approach the test vendor for a permissive mode administration, 
meaning that the security constraints have to be turned off. 


Hinkle described the process that the state had used to gather information on the use of 
speech-to-text during the state assessment. Experts in the state and other experts were 
asked about the use of speech-to-text in the classroom and for testing. These individuals 
identified many issues, including difficulties encountered in making it work in the 
classroom; test security issues in turn requiring test administration to be done in a one-on- 
one testing situation; the additional time needed to train teachers, test administrators, and 
students how to use the speech-to-text software; and increased cognitive load demands on 
the student. 


Hinkle also noted other issues. For example, even though the state’s assessment system 
has a high web accessibility rating, nothing external (which is what speech-to-text is) 
really works well with the system. Further, there are questions about whether the 
construct of writing is compromised when speech-to-text is used. Finally, if speech-to- 
text is allowed, there will be a need for a lot of professional development. 


Hinkle concluded his presentation with a set of questions that need to be answered: 

e How do we determine whether a student needs a scribe or speech-to-text? 

e Which “scribe” approach is best for a student and provides the best access? 

e How do we develop skills in using speech-to-text with students? 

e How will districts provide software and training to provide access equally for all 
learners? 

e How will we evaluate that the support provides access and does not inhibit 
performance? 

e What process will be used to evaluate whether additional services are needed? 

e How do we support districts in professional development across settings? 


State Perspective: Toni Wheeler, Washington Office of Superintendent of Public 
Instruction 

Toni Wheeler reinforced the importance of the topic by documenting the dramatic shift in 
the use of technology in the state. Cloud-based devices are being used by more than 50% 
of districts, and almost all of the rest are considering the use of these devices. Related to 
speech-to-text, there have been increases (up to 60%), and at the same time, there has 
been a decrease across years in the requests for non-standard accommodations using 
word prediction software. The state response was to have districts understand the features 
of the program and then conduct their own evaluation. This in turn produced the 
increased interest in speech-to-text. 


The state involved four groups in the discussion about a host of speech-to-text-related 
software: assessment; instructional technology; the special education technology center; 
and the state’s test vendor. Several software options were reviewed. This process 
identified many compatibility challenges, including software that initially appeared to 
work but then subsequently did not. 


Wheeler also noted the innovative approaches that districts were using to “fix” the 
incompatibility problems that arose. For example, some simply had the student use an 
unfamiliar device or software to access speech-to-text features. Others had the student 
use their classroom machine for speech-to-text, then had them access the assessment on a 
second machine, with a scribe then transcribing the student’s responses into the text 
delivery platform. Still others simply used a scribe in place of the typical software, 
application, or device. 


Wheeler concluded by noting some concerning unintended consequences of the 
challenges with speech-to-text. First, districts are using programs or software that do not 
align with the technology shifts. Second, what works for the test delivery platform is 
driving what IEP teams are selecting for students to use during instruction. 


Assessment Vendor Perspective: Jan McSorley, Pearson 

Jan McSorley noted that her years as an Assistive Technology Specialist and a special 
educator influenced her assessment vendor perspective. After defining assistive 
technology and noting the need to balance assistive technology with accessible 
educational materials, she clarified that there are two types of access tools. First, there are 
tools that are integrated and second, there are “third-party” tools. She asserted that both 
types are needed. 


McSorley indicated that speech-to-text is a complex access system that has evolved over 
time. Voice Recognition (VR) is the technology behind speech-to-text. VR has improved 
dramatically since its inception in the 1950s. The VR technology in smart devices is 
relatively accurate for short commands, but accuracy diminishes as the length of dictation 
increases. She also noted that a variety of voice recognition software is available, but 
only one of them is fully controlled by voice only; most need a mouse. 


McSorley raised numerous speech-to-text barriers for students with disabilities. These 
included cognitive, sensory, emotional, physical, and environmental barriers. She also 
noted one more barrier—policy requirements that impede access that could be provided 
by speech-to-text (e.g., requiring the student to spell every word, including indicating 
where there are spaces between words and where capitalization and punctuation are to be 
placed). 


McSorley reinforced the importance of following international accessibility guidelines as 
the foundation of universal design. The Web Content Accessibility Guidelines (WCAG) 
are international guidelines that have recently been incorporated into the revised Section 
508 statute of the United States Rehabilitation Act of 1973. She concluded her 
presentation by listing seven additional aspects of best practice: 

e Conduct user studies with students with disabilities. 

e Do quality assurance testing with assistive technology. 

e Expose developers to subject matter experts. 

e Expose developers to people with disabilities who are native users of assistive 

technology. 
e Partner with assistive technology vendors. 


e Stay abreast of the legal and technical landscapes (make no assumptions). 
e Partner with platform and browser vendors to help them understand the needs of 
your students. 


Assessment Consortium Perspective: Trish Klein, Smarter Balanced 

Trish Klein started her description of speech-to-text from an assessment consortium 
perspective by reminding participants about the technology behind the speech-to-text 
resource. Although this technology started in the 1990s, and has dramatically improved 
since then, it still is not where it needs to be. She proposed that as the technology 
becomes more mainstream, advancements will move more quickly. 


Klein highlighted the Smarter Balanced Usability, Accessibility, and Accommodations 
Guidelines, yet noted that the policies still can vary from state to state. The consortium 
has implemented a temporary approval process. She also stressed that the consortium’s 
Guidelines include a description of the speech-to-text policy and provides specific 
recommendations for use (e.g., for students with motor or processing disabilities or 
students with a temporary disability). The recommendations include the requirement that 
the student is familiar with the software and has had many opportunities to use it in daily 
instruction. 


Klein indicated that the consortium has been working on an AT readiness document to 
complement its Technology Readiness document. This guide, which will be available 
soon, addresses student preparation as well as assistive technology and information 
technology preparation. 


Klein concluded her presentation by noting the importance of collaboration for ensuring 
the success of a student who uses speech-to-text. She also noted that there needs to be 
collaborative research by assistive technology vendors, assessment vendors, districts, and 
assessment consortia. This research could look at over- and under-use of the technology. 
It also could look at the effects of the technology for all students, not just those with IEPs 
or 504 plans. Klein also reported that there is current collaboration happening between 
assistive technology vendors, assessment vendors, and assessment consortia around web- 
based applications for speech-to-text and interoperability with secure browser testing 
environments. 


Summary of Small Group Discussions 

After the rich presentations summarized here, meeting participants divided into 
discussion groups: one group discussed issues for general education students; one group 
shared perspectives on ELs; and two groups focused their discussions on students with 
disabilities. Each group targeted the conversation to address three questions: 


1. What are the implementation and policy challenges that need to be addressed? 

2. How can states work with vendors to get speech-to-text systems that more 
smoothly interface with the test delivery system? 

3. What research is needed for speech-to-text? 


The group discussions are summarized here following the question format that was used 
for the discussion. The term “speech-to-text” is used in these notes to cover all voice 
recognition approaches. 


Students in General Education 

What are the implementation and policy challenges that need to be addressed? 

The group had a rich discussion about implementation and policy challenges that need to 
be addressed. Specific implementation challenges mentioned included: 

e With the increase in use of technology as a way of life, there is likely to be an 
increase in the use of speech-to-text with general education students. 

e Speech-to-text is likely to be of benefit to general education students in many 
content areas (math, science, social studies), but teachers are saying they also use 
it in the classroom to teach language arts. 

e How to operationalize the use of speech-to-text if it really is going to be for all 
students will be a significant implementation issue. 

e Test security challenges are also likely to affect implementation. 

e Thinking about speech-to-text should be different for instruction and for 
assessment. It might be very appropriate for instruction, but not for an assessment. 

e Requesting and implementing speech-to-text is usually a very time consuming 
process; it should be made less cumbersome. 


Participants in the group also identified several policy-related challenges. For example: 

e When speech-to-text is used during an assessment, there is often a requirement 
that the student say capital letters, punctuation, etc., even when these conventions 
are not being scored. That does not make sense. 

e [tis very important to think about the construct the assessment is attempting to 
measure. If spelling is a part of what the assessment is measuring, then when 
using speech-to-text, spelling should be required. 

e Policies are set up in different ways. For example, one state policy places speech- 
to-text under the heading of “scribe.” This almost suggests that scribe is the 
preferred approach. Is this appropriate? 

e Should requirements for use of speech-to-text by general education students be 
different from what they are for students with disabilities? 


How can states work with vendors to get speech-to-text systems that more smoothly 
interface with the test delivery system? 
The group discussed a few issues that are related to states and vendors working to get 
speech-to-text systems that work more smoothly with the test delivery system. The 
following points were noted: 
e All the players should sit down at the same table to collaborate. 
e States and districts should indicate when there is a speech-to-text feature that they 
like so that the chances are increased that other vendors will create similar 
features. 
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Clarity in terminology is needed. For example, the terms “speech recognition” 
and “voice recognition” should be distinguished from speech-to-text. Speech-to- 
text takes the input (voice recognition) and then outputs text. 


What research is needed related to speech-to-text? 
Participants in the group generated several ideas for research on speech-to-text: 


Study the skills students need to use speech-to-text. 

Conduct a study to obtain the student perspective on what is needed related to 
speech-to-text. 

Study how the three tiers of accessibility and accommodations support that are 
now available impact the student experience. 

Study the effect of the speech-to-text accommodation on the construct. 

Compare experiences and outcomes when a student uses a scribe versus when the 
student uses speech-to-text. 


Students Who Are English Learners (ELs) 

What are the implementation and policy challenges that need to be addressed? 
Group participants had much to say about the implementation and policy challenges that 
need to be addressed for ELs and speech-to-text. Comments addressed implementation 
challenges for the general assessment and the assessment of English language proficiency 


(ELP): 


The speech-to-text device has to be able to use the student’s voice. 

Providing speech-to-text for an ELP assessment is different from providing it for 
the content assessment because of the constructs they measure. An ELP 
assessment measures writing in English, so speech to text that corrects English 
may not be appropriate. 

For the ELP assessment, instead of using speech-to-text, vendors are relying on 
voice recordings (for the speaking domain); this has its own challenges. 


Participants also identified policy challenges. These included: 


Policies rely on a parochial definition of writing, which affects the use of speech- 
to-text. 

Sometimes, scribe is allowed even though there is no provision for speech-to-text 
because it is not real “writing.” 

Speech-to-text is available for ELs with disabilities, but then, because of vendor 
limitations, there is a need to go to a scribe. 

Policy should be to allow ELs to use their daily devices for the assessment. 
Policy should clearly define the construct so that it can then follow with (if 
appropriate) allowing speech-to-text. 


How can states work with vendors to get speech-to-text systems that more smoothly 
interface with the test delivery system? 

The group indicated several ways that states could work with vendors on speech-to-text 
systems. In addition, they spoke about what vendors needed to do. Some of the points 
made were: 
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Vendors need to invest in upgrades to their speech-to-text systems. 

States should ask vendors for greater clarity on what will work with the test 
platform. 

Training a speech-to-text system for an EL is much more difficult than it is for 
other students due to accents. It would be great if vendors could figure out how to 
make this less difficult. 


What research is needed related to speech-to-text? 

Participants in the group generated several research ideas. There was some hesitation in 
identifying research ideas because of concerns that speech-to-text might not be best for 
ELs. Among the research ideas were: 


Explore how different speech-to-text systems work for speakers with accents due 
to their first language. 

Test out whether certain speech-to-text software works better for certain 
languages. 


Students Who Have IEPs or 504 Plans 
Because of large numbers, two groups of participants talked about students who have 
IEPs or 504 plans. The notes from the two groups have been merged here. 


What are the implementation and policy challenges that need to be addressed? 
Participants in the two groups identified numerous challenges associated with speech-to- 
text that need to be addressed. These included: 


Who really needs speech-to-text is still ambiguous. 

Easy-to-use speech-to-text tools are available for classroom instruction, yet tend 
not to be used because they are not available on the assessment. 

Students who use speech-to-text need to be responsible for making sure that what 
they said is accurately reflected in the text produced. 

It is challenging to find time to practice speech-to-text as it will appear on the 
assessment. 

Speech-to-text software can include word prediction features; it is a problem if 
this feature cannot be turned off. 

There needs to be consideration of the integrity of instruction and whether it will 
be compromised by allowing the use of speech-to-text. 

A huge implementation issue is the sheer amount of time that it takes to prepare 
students to use speech-to-text. 

There are staffing issues associated with the use of speech-to-text for assessments; 
students often have to be tested one-on-one if they are using speech-to-text. 
Security continues to be an implementation issue. 

Small school districts may need extra support if they are to implement speech-to- 
text. 


Among the policy-related comments that participants in the two groups addressing 
students who have IEPs or 504 plans were the following: 
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Policy is defining the construct, but may be doing so in a way that disadvantages 
the real meaning of writing, which is so much more than putting words on paper. 
It should be about the ideas. 

Educators often confuse speech-to-text with text-to-speech. Policies should be 
explicit and provide definitions of terms. 


How can states work with vendors to get speech-to-text systems that more smoothly 
interface with the test delivery system? 

Participants made numerous comments on the topic of states working with vendors on the 
interface of speech-to-text with the test delivery system. Their comments included: 


States need to ensure that their assessment systems allow for the use of third-party 
software for speech-to-text. 

States should work with vendors to create profiles for those students who will use 
speech-to-text so that certain aspects of assistive technology cannot be used by the 
student during the assessment. 

States should work with vendors to create a glossary of terms related to speech-to- 
text (e.g., word prediction, predict ahead, voice recognition, etc.). 

Address what sometimes seems to be a conflict between assistive technology staff 
and information technology staff; this seems like a conflict between accessibility 
and security. This means communication is needed between states and vendors 
because assessment vendors tend to talk to information technology staff, not 
assistive technology staff. 

States need to share changing accessibility policies with vendors; this is 
sometimes done too late (or never), which means that the vendor cannot adjust the 
technology to match the policy. 

Collaboration among states and vendors is needed to provide the best speech-to- 
text for students. 

Figure out a way to pilot the use of software in a sample of districts of all sizes (to 
check bandwidth issues, testing concerns, etc.). 


What research is needed related to speech-to-text? 
Numerous ideas by the two groups addressing students with IEPs and 504 plans included: 


Gather data on how many students actually use speech-to-text. 

Conduct cognitive labs to help identify all the problems that emerge in the 
implementation of speech-to-text. 

Hold focus groups with teachers (or interview teachers) to obtain a deeper 
understanding of how they balance using speech-to-text and writing instruction. 
Study whether the cognitive load demands of speech-to-text change with the 
demands of the questions. 

Document the actual use of speech-to-text rather than requests for it. 

Compare implementation issues that arise for scribe versus for speech-to-text. 
Ask students about their experiences when using speech-to-text in the classroom 
and on assessments. 

Conduct research on the areas of need that are addressed by speech-to-text (e.g., 
legibility of handwriting, processing issues, voicing math, etc.). 
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Conclusion 

The meeting closed with debriefs from each of the groups, and concluding remarks by 
Martha Thurlow, Director at NCEO. She thanked participants for their thoughtful 
discussion and confirmed that a report on the pre-session would be available before the 
end of the summer. 
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